Complex-Valued Phase-Based Eulerian Motion Modulation

ABSTRACT

In one embodiment, a method of amplifying temporal variation in at least two images includes converting two or more images to a transform representation. The method further includes, for each spatial position within the two or more images, examining a plurality of coefficient values. The method additionally includes calculating a first vector based on the plurality of coefficient values. The first vector can represent change from a first image to a second image of the at least two images describing deformation. The method also includes modifying the first vector to create a second vector. The method further includes calculating a second plurality of coefficients based on the second vector.

RELATED APPLICATION(S)

This application is a continuation-in-part of U.S. application Ser. No.13/607,173, filed Sep. 7, 2012.

BACKGROUND OF THE INVENTION

Phase-Based Optical Flow.

Phase-based optical flow can track constant phase contours by computingthe phase gradient of a spatio-temporally bandpassed video. This canprovide an approximation to the motion field, and it was shown thatphase is more robust than amplitude in detecting image changes due tocontrast and scale.

Complex Steerable Pyramids.

Steerable pyramids are over complete representations of images thatdecompose the images along scale and orientation. The basis functionsare similar to Gabor wavelets, which are sinusoids multiplied by asmooth spatial envelope.

A steerable pyramid can be real-valued by having all real coefficients,but it can be extended to complex-valued coefficients where thesinusoidal basis functions are replaced by a complex exponential. Inother words, the real part represents an even-symmetric filter (cosinephase), while its imaginary counterpart represents an odd-symmetricfilter (sine phase). While the real and imaginary parts increase theover completeness of the representation, the complex-valued steerablepyramid is a richer representation that separates the amplitude of thelocal wavelet from its phase, allowing for the convenient measurement oflocal phase information.

SUMMARY OF THE INVENTION

The present method builds on the link between phase and motion, butavoids the explicit computation of flow vectors, and instead directlymanipulates phase variations in videos. The present method furtheremploys the phase component to analyze motion.

In one embodiment, a method of amplifying temporal variation in at leasttwo images includes examining pixel values of the at least two images.The temporal variation of the pixel values between the at least twoimages is below a particular threshold. The method then applies signalprocessing to the pixel values.

In another embodiment, a method of amplifying temporal variation in atleast two images includes converting two or more images to a transformrepresentation. The method further includes, for each spatial positionwithin the two or more images, examining a plurality of coefficientvalues. The method additionally includes calculating a first vectorbased on the plurality of coefficient values. The first vector canrepresent change from a first image to a second image of the at leasttwo images describing deformation. The method also includes modifyingthe first vector to create a second vector. The method further includescalculating a second plurality of coefficients based on the secondvector.

In another embodiment, the transform representation is a pyramid. Thepyramid can be a complex steerable value pyramid. The first and secondpluralities of coefficient values can include real and imaginary partsof the coefficients.

In one embodiment, the first vector can be a scalar. Modifying the firstvector to create the second vector can multiply the scalar by aconstant. The first vector represents a change in phase from a firstimage of the at least two images and the second image of the at leasttwo images.

In another embodiment, the method can also include generating at leastone image based on the second plurality of coefficients.

In another embodiment, the two or more images may be a set of two ormore images.

In a further embodiment, the method may include temporally bandpassingthe second vector.

In one embodiment, a system for amplifying temporal variation in atleast two images can include a transform module configured to converttwo or more images to a transform representation. The system can furtherinclude a coefficient examination module configured to, for each spatialposition within the two or more images, examine a plurality ofcoefficient values. The system can further include a vector calculationmodule configured to calculate a first vector based on the plurality ofcoefficient values. The first vector can represent change from a firstimage to a second image of the at least two images describingdeformation. The system can additionally include a vector modificationmodule configured to modify the first vector to create a second vector.The system can also include a calculation module configured to calculatea second plurality of coefficients based on the second vector.

The embodiments of the system and method described above can be referredto as a “phase-based” system and method.

BRIEF DESCRIPTION OF THE DRAWINGS

The foregoing will be apparent from the following more particulardescription of example embodiments of the invention, as illustrated inthe accompanying drawings in which like reference characters refer tothe same parts throughout the different views. The drawings are notnecessarily to scale, emphasis instead being placed upon illustratingembodiments of the present invention.

FIGS. 1A-B are images illustrating motion magnification of a sourcevideo of a baby breathing.

FIG. 2A illustrates an input frame from a video showing a membraneoscillating in response to a sound wave played by a speaker.

FIG. 2B illustrates ideal transfer functions of different filters in acomplex steerable pyramid with three scales and two orientation bands,drawn on the spatial frequency plane.

FIG. 2C illustrates phase videos corresponding to different scales andorientations in FIG. 2B.

FIG. 2D illustrates observations of phase over time for a singlelocation.

FIG. 3 illustrates a comparison of phase-based motion magnification withlinear motion magnification for the case of a sinusoidal spatial signal.

FIG. 4 illustrates diagrams showing the phase-based embodiment achievingperformance being at least twice that of the linear embodiment withoutsuffering from clipping artifacts for general non-periodic structures.

FIG. 5A illustrates a frame in a sequence of IID noise.

FIG. 5B illustrates motion amplified by an amplification factor, afteremploying the linear method.

FIG. 5C illustrates motion amplified by an amplification factor, afteremploying the phase-based method.

FIG. 5D illustrates a plot of the error as function of noise for eachmethod, using several magnification factors.

FIG. 6 illustrates representative frames from videos used to demonstratethe methods described herein.

FIG. 7 is a flow diagram illustrating an example of embodiment of thephase-based method described in this application.

FIG. 8 illustrates a computer network or similar digital processingenvironment in which the present invention may be implemented.

FIG. 9 is a diagram of the internal structure of a computer (e.g.,client processor/device or server computers) in the computer system ofFIG. 8.

DETAILED DESCRIPTION OF THE INVENTION

A description of example embodiments of the invention follows.

FIGS. 1A-B are images 100 and 140 illustrating motion magnification of asource video of a baby breathing. FIG. 1A shows a frame 100 from aninput sequence (baby), with a spatiotemporal vertical-line-versus-time(YT) slice 110 taken at the location marked by vertical line 102. FIG.1B shows a motion magnified frame 140 based on the input sequence withspatiotemporal YT slices 150 and a zoom-in of motion 144 on a patcharound the baby's chest in the insets. The magnification methodillustrated by FIG. 1B is a phase-based magnification. The magnificationmethod illustrated by FIG. 1B supports larger magnification factors withless artifacts and noise compared to other methods.

The method and apparatus described herein magnifies and reveals smallmovements in videos based on a novel analysis of motion employingcomplex-valued image pyramids. Instead of employing computation ofoptical flow or motion vectors that Lagrangian methods require, themethod shows that the variation of the phase of the coefficients of acomplex-valued steerable pyramid over time corresponds to motion and canbe temporally enhanced or modulated. The method is fast, and is lesssensitive to noise than previous Eulerian approaches.

Many phenomena exhibit motions that are too small to be well perceivedby the naked eye, and require computational amplification to berevealed. Lagrangian approaches compute velocity and warp frames of thevideo according to magnified velocity vectors, which additionallyrequires background inpainting. One Eulerian approach alleviates theneed for costly flow computation, and performs processing in a separablemanner in space and time. Eulerian video processing can also be employedto dampen temporal aliasing of motion in videos. Unfortunately, typicalEulerian video magnification supports relatively small magnificationfactors for high spatial frequencies, and tends to significantly amplifynoise when the magnification factor is increased.

To solve these problems, the method presents a new Eulerian approach tomotion magnification based on complex-valued steerable pyramids,inspired by phase-based optical flow and motion without movement. Thelocal phase variation over time of the complex-valued steerable pyramidcoefficients corresponds to motion. After computing this phasevariation, temporal processing can amplify motion in selected temporalfrequency bands and reconstruct a modified video.

The link between motion and phase in steerable pyramids can be analyzedto determine limits of the method, which are linked to the spatialsupport of the steerable basis functions. The method improves on theprevious Eulerian magnification method in two ways:

1) The method achieves twice the magnification of prior methods intheory and is at least as good in practice; and

2) The method has substantially better noise performance. Prior methodsamplify noise linearly because they amplify temporal brightness changes.In contrast, the present method modifies phase, not amplitude, whichtranslates noise without increasing its magnitude. The present methodcan achieve a greater degree of motion magnification while producingfewer artifacts in similar running times. Moreover, as the methodsupports larger degrees of motion modulation, it can further enable newapplications for video manipulation such as motion fast forward, andmotion pause.

FIG. 2A illustrates an input frame 200 from a video showing a membrane204 oscillating in response to a 110Hz sound wave played by a speaker202.

FIG. 2B illustrates ideal transfer functions 210 of some differentfilters in a complex steerable pyramid with three scales and twoorientation bands, drawn on the spatial frequency plane (high and lowpass residuals 212 in medium gray; black pixels 214 indicate spatialfrequencies not modulated by the transfer functions of the steerablepyramid).

FIG. 2C illustrates phase videos 220 corresponding to different scalesand orientations in FIG. 2B. FIG. 2D illustrates observations of phaseover time for a single location 222 (FIG. 2C). Scale and orientation 230reveals motions at the frequency of interest 240 (e.g., 110 Hz). Bytemporally filtering and amplifying the phase data, the method magnifiesthe motions in a required frequency range.

The steerable pyramid has high and low pass residuals that do not haveorientation and are real-valued. The high and low pass residuals areidentical to the corresponding levels in a real steerable pyramid. Thetransfer functions in the oriented bands of the steerable pyramidΨ_(ω,θ) are scaled, rotated copies indexed by scale ω and orientation θ.The steerable pyramid is built by applying these transfer functions tothe Fourier transform {hacek over (I)} of an image I to decompose itinto different spatial frequency bands S_(ω,θ)={hacek over (I)}Ψ_(ω,θ).That is, the Fourier transform of each pyramid sub-band is obtained bymultiplying the FFT of the image by a “mask” similar to those in FIG.2B, which selects a range of frequencies and orientation. FIG. 2B showsthat the positive frequency values ω are retained and that theirsymmetric −ω values are zeroed out. This is why, when the masked FFT isinverse-transformed back to the image domain, the corresponding sub-bandhas complex values. Then, we utilize the real component of theinverse-transformed images.

In the frequency domain, the processing to build and then collapse thepyramid yields the reconstructed image, Ĩ_(r), which is given by

Ĩ _(R) =ΣS _(ω,θ)Ψ_(ω,θ) =ΣĨΨ _(ω,θ) ²   (1)

where the sums are over all of the scales and orientations in thepyramid. The processing is done in the frequency domain, but the methodstores and uses the complex-valued results in the primal domain.

The transfer functions of a complex steerable pyramid only contain thepositive frequencies of a real steerable pyramid's filter. That is, theresponse of 2 cos(ωx)=e^(iωx)+e^(−iωx) is e^(iωx) (see FIG. 2). Thetransfer functions mask regions, so that they are capable of isolating arange of frequencies as opposed to a single localized frequency.

Phase-Based Motion Magnification

The present method computes the local phase at every scale andorientation. Then, the method temporally bandpasses these phases toisolate specific temporal frequencies and in particular to remove the DCcomponent. These temporally bandpassed phases correspond to motion indifferent scales and orientations. To synthesize magnified motion, themethod multiplies the temporally bandpassed phases by an amplificationfactor α. The method then uses these amplified phase differences tomagnify (or decrease) the motion in the sequence by modifying the phaseof each frame by this amount.

Motion Modulation

In one example, the method analyzes a one-dimensional (1D) imageintensity profile f under pure translation over time, f(x+δ(t)), forsome displacement function δ(t). The method attempts to synthesize amotion-magnified sequence, f(x+(1+α)δ(t)), for the amplification factorα.

Linear Video Magnification

First, a linear magnifying a video applies a temporal filter to theintensity sequence. Assuming a DC balanced broadband temporal filter,this results in

$\begin{matrix}{{B\left( {x,t} \right)} = {{\delta (t)}\frac{\partial f}{\partial x}}} & (2)\end{matrix}$

which is then amplified and added back to the original frames. Thisleads to the following first-order (linear) approximation of themotion-magnified signal:

$\begin{matrix}{{{f(x)} + {\left( {1 + \alpha} \right){\delta (t)}\frac{\partial f}{\partial x}}} \approx {f\left( {x + {\left( {1 + \alpha} \right){\delta (t)}}} \right)}} & (3)\end{matrix}$

This approximation holds for small values of α and δ(t) and low spatialfrequencies, but breaks down quickly and causes artifacts (e.g., noise).

Phase-Based Video Magnification

An embodiment of the phase-based method relies on complex-valuedsteerable pyramids because they afford a local notion of phase thatallows direct shifting of the local Gabor-like wavelets that representthe image. The phase-based method employs a Fourier expression of thedisplaced image profile f(x+δ(t)) as a sum of complex sinusoids.

$\begin{matrix}{{f\left( {x + {\delta (t)}} \right)} = {\sum\limits_{\omega = {- \infty}}^{\infty}{A_{\omega}^{{\omega}{({x + {\delta {(t)}}})}}}}} & (4)\end{matrix}$

For this analysis, the phase-based method approximates the pyramidconstruction and assumes that the Fourier masks in FIG. 2B select asingle frequency ω and angle θ per pyramid orientation band. Startingfrom Eq. 4, the primal-domain idealized pyramid level for frequency ω is

S _(ω,θ)(x,t)=A _(ω) e ^(iω(x+δ(t)))   (5)

FIG. 3 illustrates phase-based motion magnification for the case ofsinusoidal functions. In these plots, the initial displacement isδ(t)=1. True amplification 300 is illustrated compared to amplificationusing a linear method 310 and the phase-based embodiment 320. While theerrors for the technique of the linear method 310 of the method aredependent on wavelength for sinusoids, any errors in graph 330 from thephase-based embodiment 320 of the method are not dependent on wavelengthand the error is uniformly small. The phase-based errors curves 338 aresuperimposed about zero error in graph 330, along the x-axis. On theother hand, linear-based error curves 332, 334, and 336 are shown to begreater for lower wavelengths, and only become close to zero as thewavelength becomes larger.

The pixels of sub-band S_(ω,θ) have complex values and their magnitudesencode the amplitude of the sine wave, while their complex phasesdepends on the pixel and vary according to the translating motion.

The phase-based method manipulates the phase of the bands of thepyramid. The phase-based method makes derivations simple by applying azero-DC broadband temporal filter to the phase components, ω(x+δ(t)),for which the response for every scale and orientation is

B _(ω,θ)(x,t)=ωδ(t)   (6)

The phase-based method multiplies the bandpassed phase by anamplification factor α, takes a complex exponential, and multiplies theresult by the corresponding scale and orientation band, to get

$\quad\begin{matrix}\begin{matrix}{{{\hat{S}}_{\omega,\theta}\left( {x,t} \right)} = {{S_{\omega,\theta}\left( {x,t} \right)}^{{\alpha}\; B_{\omega,\theta}}}} \\{= {A_{\omega}^{{\omega}{({x + {{({1 + \alpha})}{\delta {(t)}}}})}}}}\end{matrix} & (7)\end{matrix}$

The phase-based method receives the analytic signal at each scale andorientation, phase-shifted by (1+α)δ(t). The phase-based embodimentobtains the output motion-magnified sequence by taking the real part ofEq. 7 recovering the negative spatial frequencies, and collapsing thesteerable pyramid, which approximates f(x+(1+α)δ(t)). FIG. 3 illustratesthat the phase-based method is nearly exact for sinusoidal waves, assuch signals contain a single spatial frequency.

A linear method is exact in the case of linear ramps while thephase-based embodiment of the method is nearly exact for sinusoidalwaves (FIG. 3). However, both the linear method and phase-based methodrely on spatial pyramids, where each level is bandlimited. Suchspatially bandpassed images are better approximated by sinusoidal wavesthan linear ramps.

FIG. 4 illustrates diagrams 400 and 410 showing the phase-based methodachieving performance being at least twice that of a linear methodwithout suffering from clipping artifacts for general non-periodicstructures. Wrap-around artifacts may become noticeable when structuresare phase-shifted through 2π for large amplification (e.g., diagram410).

Limits on the Magnification

As a rule of thumb, if the spatial wavelength lambda, the magnificationalpha, and the original motion displacement delta(t) satisfy thisrelationship:

$\begin{matrix}{{{\alpha\delta}(t)} < \frac{\lambda}{4}} & (8)\end{matrix}$

then the motion magnification gives accurate and visually appealingresults.

Sensitivity to Noise

Phase-based motion magnification has excellent noise characteristics. Asthe amplification factor is increased, noise is translated rather thanamplified. At a particular scale and orientation band, the response fora noisy image I+σn might be

S _(ω,θ) =e ^(iω(x+δ(t))) +σN _(ω,θ)(x,t)   (9)

σ is much lower in magnitude than the noiseless signal, so that temporalfiltering of the phase is approximately ωδ(t) as in Eq. 5. To magnifythe motion, the response is shifted by e^(iαωδ(t)), so that the motionmagnified band is

Ŝ _(ω,θ) =e ^(iω(x+(1+α)δ(t))) +σe ^(iαωδ(t)) N _(ω,θ)(x,t)   (15)

The only change to the noise after processing is a phase shift. When thepyramid is collapsed, this phase shift corresponds to a translation ofthe noise. In contrast, a linear method amplifies the noise linearly inproportion to α.

Results

In the phase-based method, all processing is performed using a complexsteerable pyramid with scales that are an octave apart and fourorientations. Complex steerable pyramid code computes the filterresponses in the frequency domain. Processing is performed in YIQ colorspace, so that the amplification factor could be attenuated in thechrominance channels. It took an order of a few minutes to process a512×512 pixel video with 300 frames using non-optimized MATLAB code on amachine with 12 cores and 64 GB of RAM. It can be efficientlyimplemented to run in real time similar to the linear embodiment ascomputing a steerable—rather than Laplacian—decomposition introducesonly a minor performance overhead. The user has control over theamplification factor and the temporal bandpass filter.

FIGS. 5A-D illustrate comparisons between linear and phase-basedEulerian motion magnification in handling noise. FIG. 5A illustrates aframe 600 in a sequence of IID noise. FIGS. 5B-C illustrate motion 610and 620 amplified by a factor of 50, where FIG. 5B employs the linearmethod and FIG. 5C employs the phase-based method. FIG. 5D illustrates aplot 630 of the by showing root-mean-square deviation (RMSD) as asfunction of noise for each method, using several magnification factors,showing the advantages of the proposed phase based approach versus thelinear based approach. For example, the RMSE 632, 634, and 636 fromlinear methods all increase at a rate faster than the RMSE 638 from the

The phase-based method can be applied to a set of natural and syntheticvideos, and the results can be compared with the some of them ones bythe linear embodiment. The results on baby (FIG. 1), and guitar andcamera (FIG. 6) confirm our noise analysis and derived bounds onamplification. In particular, the magnified motions in the results ofthe phase-based method (e.g. the respiratory motions of the baby andvibrating guitar strings) looks crispier, and contain significantly lessartifacts and noise compared to a linear method. At the cost of moderateartifacts, the phase-based embodiment can magnify the motion of cameradramatically more than previous Eulerian techniques.

The phase-based embodiment can magnify microsaccades, which are subtle,involuntary, low amplitude (10-400 micron) movements in the human eye.The sample video of the microsaccades is taken with a high speed cameraat 500 Hz. The phase-based embodiment processed a one second (500 frame)sequence with an ideal bandpass signal with passband between 30 Hz and50 Hz. The phase-based embodiment applies a spatial mask to the phaseshifts to emphasize the motion around the iris. Such a detection systemcan have clinical applications, as the frequency content of ocularmicrotremor can have clinical significance.

FIG. 6 illustrates representative frames 700, 710, 720, 730, 740 and 750from videos used to demonstrate the methods.

In another demonstration, a tense membrane is made of a sheet of rubbermounted on a section of PVC pipe using a rubber band. A loudspeakervibrates air that in turn vibrates the membrane. A high speed cameracaptures the result. The membrane has two modes when waveforms at 76 Hzand 110 Hz are sent through the loudspeaker. A video of the membrane iscaptured when a composite waveform of these two frequencies are playedthrough the loudspeaker. The phase-based method separates and amplifiesthese two modes.

As the embodiments support large amplification factors, a user canincrease the amplification, α, with time to simulate motion fastforwarding. This creates the illusion of speeding up time. For example,the phase-based embodiment can speed up motion of a shadow moving overtime and show what would happen if the sequence had been recorded for alonger period of time.

The phase-based method also allows pausing motions in videos. Thephase-based method can remove low amplitude short term motions fromvideos while larger amplitude motions continue to pass through. This issimilar to motion denoising and video de-animation, but can be done inreal-time in the phase-based method. To pause the motion in a sequence,the phase-based embodiment computes the phases for a single referenceframe and sets the phases in the entire video equal to that of thereference frame. The result is not the same as a constant referenceframe as the coefficient amplitudes are still evolving over time. Forexample, the phase-based method can remove the motion of the subway carwhile the car passes through the tunnel. In another example, thephase-based method can remove the rustling of the leaves while theillumination in the scene changes.

The method can also amplify color variations by temporally bandpassingthe amplitude (Eg. 4) and the low-pass residual of the steerablepyramid. This yields similar results to the linear method for extractingand visualizing the human pulse signal, since in both cases the sameprocessing is applied to the low-pass residual band of the image pyramid(e.g., a Laplacian pyramid in the case of the linear method, and asteerable pyramid in the case of the phase-based method).

Lagrangian approaches to motion magnification can amplify the motion ina video arbitrarily, but rely on accurate optical flow estimates, imagesegmentation, and inpainting. Like the Lagrangian approach, thephase-based method translates structures within the image. However, itdoes not involve lengthy computation and it can run in real time. Theapproach is limited by the fact that structures cannot be translatedbeyond the spatial envelopes of the steerable pyramid. High frequenciescannot be translated as far as low frequencies and this leads structureto break up as the low frequency components translate beyond the highfrequency ones.

FIG. 7 is a flow diagram 900 illustrating an example of embodiment ofthe phase-based method described in this application. The phase-basedmethod begins by converting two or more images to a transformrepresentation (902). Then, the method examines coefficient values foreach spatial position in the two or more images (904). The method thencalculates a first vector based on the coefficient values (906). Then,the method modifies the first vector to create a second vector (908).The second vector represents amplified movement between the two images.The method then calculates a second set of coefficient values based onthe second vector (910). Then, the method generates an image based onthe second set of coefficients (912). This generated image should havethe motion amplified according to the second vector.

FIG. 8 illustrates a computer network or similar digital processingenvironment in which the present invention may be implemented.

Client computer(s)/devices 1050 and server computer(s) 1060 provideprocessing, storage, and input/output devices executing applicationprograms and the like. Client computer(s)/devices 1050 can also belinked through communications network 1075 to other computing devices,including other client devices/processes 1050 and server computer(s)1060. Communications network 1075 can be part of a remote accessnetwork, a global network (e.g., the Internet), a worldwide collectionof computers, Local area or Wide area networks, and gateways thatcurrently use respective protocols (TCP/IP, Bluetooth, etc.) tocommunicate with one another. Other electronic device/computer networkarchitectures are suitable.

FIG. 9 is a diagram of the internal structure of a computer (e.g.,client processor/device 1050 or server computers 1060) in the computersystem of FIG. 8. Each computer 1050, 1060 contains system bus 1179,where a bus is a set of hardware lines used for data transfer among thecomponents of a computer or processing system. Bus 1179 is essentially ashared conduit that connects different elements of a computer system(e.g., processor, disk storage, memory, input/output ports, networkports, etc.) that enables the transfer of information between theelements. Attached to system bus 1179 is I/O device interface 1182 forconnecting various input and output devices (e.g., keyboard, mouse,displays, printers, speakers, etc.) to the computer 1050, 1060. Networkinterface 1186 allows the computer to connect to various other devicesattached to a network (e.g., network 1075 of FIG. 10). Memory 1190provides volatile storage for computer software instructions 1192 anddata 1194 used to implement an embodiment of the present invention(e.g., motion magnification code detailed above). Disk storage 1195provides non-volatile storage for computer software instructions 1192and data 1194 used to implement an embodiment of the present invention.Central processor unit 1184 is also attached to system bus 1179 andprovides for the execution of computer instructions.

In one embodiment, the processor routines 1192 and data 1194 are acomputer program product (generally referenced 1192), including acomputer readable medium (e.g., a removable storage medium such as oneor more DVD-ROM's, CD-ROM's, diskettes, tapes, etc.) that provides atleast a portion of the software instructions for the invention system.Computer program product 1192 can be installed by any suitable softwareinstallation procedure, as is well known in the art. In anotherembodiment, at least a portion of the software instructions may also bedownloaded over a cable, communication and/or wireless connection. Inother embodiments, the invention programs are a computer programpropagated signal product 1070 embodied on a propagated signal on apropagation medium (e.g., a radio wave, an infrared wave, a laser wave,a sound wave, or an electrical wave propagated over a global networksuch as the Internet, or other network(s)). Such carrier medium orsignals provide at least a portion of the software instructions for thepresent invention routines/program 92.

In alternate embodiments, the propagated signal is an analog carrierwave or digital signal carried on the propagated medium. For example,the propagated signal may be a digitized signal propagated over a globalnetwork (e.g., the Internet), a telecommunications network, or othernetwork. In one embodiment, the propagated signal is a signal that istransmitted over the propagation medium over a period of time, such asthe instructions for a software application sent in packets over anetwork over a period of milliseconds, seconds, minutes, or longer. Inanother embodiment, the computer readable medium of computer programproduct 1192 is a propagation medium that the computer system 1050 mayreceive and read, such as by receiving the propagation medium andidentifying a propagated signal embodied in the propagation medium, asdescribed above for computer program propagated signal product.

Generally speaking, the term “carrier medium” or transient carrierencompasses the foregoing transient signals, propagated signals,propagated medium, storage medium and the like.

While this invention has been particularly shown and described withreferences to example embodiments thereof, it will be understood bythose skilled in the art that various changes in form and details may bemade therein without departing from the scope of the inventionencompassed by the appended claims.

1. A method of amplifying temporal variation in at least two images, themethod comprising: converting two or more images to a transformrepresentation; for each spatial position within the two or more images,examining a plurality of coefficient values; calculating a first vectorbased on the plurality of coefficient values, the first vectorrepresenting change from a first image to a second image of the at leasttwo images describing deformation; modifying the first vector to createa second vector representing amplified movement between the first andsecond images; calculating a second plurality of coefficients based onthe second vector; and from the second plurality of coefficients,generating an output image showing motion amplified according to thesecond vector and alternatively amplified color variation between thefirst and second images.
 2. The method of claim 1, wherein the transformrepresentation is a pyramid.
 3. The method of claim 2, wherein thepyramid is a complex steerable value pyramid.
 4. The method of claim 1,wherein the first and second pluralities of coefficient values includereal and imaginary parts of the coefficients.
 5. The method of claim 1,wherein the first vector is a scalar, and wherein modifying the firstvector to create the second vector multiplies the scalar by a constant.6. The method of claim 1, wherein the first vector represents a changein phase from a first image of the at least two images and the secondimage of the at least two images.
 7. (canceled)
 8. The method of claim1, wherein the two or more images are a set of two or more images. 9.The method of claim 1, further comprising temporally bandpassing thesecond vector resulting in amplifying color variation between images.10. A system for amplifying temporal variation in at least two images,the system comprising: a transform module configured to convert two ormore images to a transform representation; a coefficient examinationmodule configured to, for each spatial position within the two or moreimages, examine plurality of coefficient values; a vector calculationmodule configured to calculate a first vector based on the plurality ofcoefficient values, the first vector representing change from a firstimage to a second image of the at least two images describingdeformation; a vector modification module configured to modify the firstvector to create a second vector representing amplified movement betweenthe first and second images; a calculation module configured tocalculate a second plurality of coefficients based on the second vector;and a modified image generation module configured to generate at leastone image based on the second plurality of coefficients, the generatedat least one image showing amplified motion according to the secondvector and alternatively amplified color variation between the first andsecond images.
 11. The system of claim 10, wherein the transformrepresentation is a pyramid.
 12. The system of claim 11, wherein thepyramid is a complex steerable value pyramid.
 13. The system of claim10, wherein the first and second pluralities of coefficient valuesinclude real and imaginary parts of the coefficients.
 14. The system ofclaim 10, wherein the first vector is a scalar, and wherein the vectormodification module is further configured to modify the first vector tocreate the second vector by multiplying the scalar by a constant. 15.The system of claim 10, wherein the first vector represents a change inphase from a first image of the at least two images and the second imageof the at least two images.
 16. (canceled)
 17. The system of claim 10,wherein the two or more images are a set of two or more images.
 18. Thesystem of claim 10, further comprising a bandpass module configured totemporally bandpass the second vector resulting in amplifying colorvariations between images.
 19. The method of claim 9 further comprisingextracting and visualizing a human pulse signal from the images.
 20. Thesystem of claim 18 wherein the bandpass module is further configured toextract and visualize human pulse signals from the images.
 21. A methodof amplifying temporal variation in at least two images, the methodcomprising: examining pixel values of the at least two images, thetemporal variation of the pixel values between the at least two imagesbeing below a particular threshold; applying signal processing to thepixel values; and temporally bandpassing pixel values to amplify colorvariation between images.
 22. The method of claim 21 further comprisingextracting and visualizing human pulse signal from the images.
 23. Asystem for amplifying temporal variation in at least two images, thesystem comprising: a pixel examination module configured to examinepixel values of the at least two images, the temporal variation of thepixel values between the at least two images being below a particularthreshold; a signal processing module configured to apply signalprocessing to the pixel values; wherein the signal processing moduletemporally bandpasses the pixel values to amplify color variationsbetween the images.
 24. The system as claimed in claim 23 wherein thesignal processing module is configured to extract and visualize humanpulse signal from the images.